Propagating Confidences through CNNs for Sparse Data Regression

In most computer vision applications, convolutional neural networks (CNNs) operate on dense image data generated by ordinary cameras. Designing CNNs for sparse and irregularly spaced input data is still an open problem with numerous applications in autonomous driving, robotics, and surveillance. To tackle this challenging problem, we introduce an algebraically-constrained convolution layer for CNNs with sparse input and demonstrate its capabilities for the scene depth completion task. We propose novel strategies for determining the confidence from the convolution operation and propagating it to consecutive layers. Furthermore, we propose an objective function that simultaneously minimizes the data error while maximizing the output confidence. Comprehensive experiments are performed on the KITTI depth benchmark and the results clearly demonstrate that the proposed approach achieves superior performance while requiring three times fewer parameters than the state-of-the-art methods. Moreover, our approach produces a continuous pixel-wise confidence map enabling information fusion, state inference, and decision support.Comment: To appear in the British Machine Vision Conference (BMVC2018

Confidence Propagation through CNNs for Guided Sparse Depth Regression

Generally, convolutional neural networks (CNNs) process data on a regular grid, e.g. data generated by ordinary cameras. Designing CNNs for sparse and irregularly spaced input data is still an open research problem with numerous applications in autonomous driving, robotics, and surveillance. In this paper, we propose an algebraically-constrained normalized convolution layer for CNNs with highly sparse input that has a smaller number of network parameters compared to related work. We propose novel strategies for determining the confidence from the convolution operation and propagating it to consecutive layers. We also propose an objective function that simultaneously minimizes the data error while maximizing the output confidence. To integrate structural information, we also investigate fusion strategies to combine depth and RGB information in our normalized convolution network framework. In addition, we introduce the use of output confidence as an auxiliary information to improve the results. The capabilities of our normalized convolution network framework are demonstrated for the problem of scene depth completion. Comprehensive experiments are performed on the KITTI-Depth and the NYU-Depth-v2 datasets. The results clearly demonstrate that the proposed approach achieves superior performance while requiring only about 1-5% of the number of parameters compared to the state-of-the-art methods.Comment: 14 pages, 14 Figure

Uncertainty-Aware CNNs for Depth Completion: Uncertainty from Beginning to End

The focus in deep learning research has been mostly to push the limits of prediction accuracy. However, this was often achieved at the cost of increased complexity, raising concerns about the interpretability and the reliability of deep networks. Recently, an increasing attention has been given to untangling the complexity of deep networks and quantifying their uncertainty for different computer vision tasks. Differently, the task of depth completion has not received enough attention despite the inherent noisy nature of depth sensors. In this work, we thus focus on modeling the uncertainty of depth data in depth completion starting from the sparse noisy input all the way to the final prediction. We propose a novel approach to identify disturbed measurements in the input by learning an input confidence estimator in a self-supervised manner based on the normalized convolutional neural networks (NCNNs). Further, we propose a probabilistic version of NCNNs that produces a statistically meaningful uncertainty measure for the final prediction. When we evaluate our approach on the KITTI dataset for depth completion, we outperform all the existing Bayesian Deep Learning approaches in terms of prediction accuracy, quality of the uncertainty measure, and the computational efficiency. Moreover, our small network with 670k parameters performs on-par with conventional approaches with millions of parameters. These results give strong evidence that separating the network into parallel uncertainty and prediction streams leads to state-of-the-art performance with accurate uncertainty estimates.Comment: CVPR2020 (8 pages + supplementary

Hinge-Wasserstein: Mitigating Overconfidence in Regression by Classification

Modern deep neural networks are prone to being overconfident despite their drastically improved performance. In ambiguous or even unpredictable real-world scenarios, this overconfidence can pose a major risk to the safety of applications. For regression tasks, the regression-by-classification approach has the potential to alleviate these ambiguities by instead predicting a discrete probability density over the desired output. However, a density estimator still tends to be overconfident when trained with the common NLL loss. To mitigate the overconfidence problem, we propose a loss function, hinge-Wasserstein, based on the Wasserstein Distance. This loss significantly improves the quality of both aleatoric and epistemic uncertainty, compared to previous work. We demonstrate the capabilities of the new loss on a synthetic dataset, where both types of uncertainty are controlled separately. Moreover, as a demonstration for real-world scenarios, we evaluate our approach on the benchmark dataset Horizon Lines in the Wild. On this benchmark, using the hinge-Wasserstein loss reduces the Area Under Sparsification Error (AUSE) for horizon parameters slope and offset, by 30.47% and 65.00%, respectively

Uncertainty-Aware Convolutional Neural Networks for Vision Tasks on Sparse Data

Early computer vision algorithms operated on dense 2D images captured using conventional monocular or color sensors. Those sensors embrace a passive nature providing limited scene representations based on light reflux, and are only able to operate under adequate lighting conditions. These limitations hindered the development of many computer vision algorithms that require some knowledge of the scene structure under varying conditions. The emergence of active sensors such as Time-of-Flight (ToF) cameras contributed to mitigating these limitations; however, they gave a rise to many novel challenges, such as data sparsity that stems from multi-path interference, and occlusion. Many approaches have been proposed to alleviate these challenges by enhancing the acquisition process of ToF cameras or by post-processing their output. Nonetheless, these approaches are sensor and model specific, requiring an individual tuning for each sensor. Alternatively, learning-based approaches, i.e., machine learning, are an attractive solution to these problems by learning a mapping from the original sensor output to a refined version of it. Convolutional Neural Networks (CNNs) are one example of powerful machine learning approaches and they have demonstrated a remarkable success on many computer vision tasks. Unfortunately, CNNs naturally operate on dense data and cannot efficiently handle sparse data from ToF sensors. In this thesis, we propose a novel variation of CNNs denoted as the Normalized Convolutional Neural Networks that can directly handle sparse data very efficiently. First, we formulate a differentiable normalized convolution layer that takes in sparse data and a confidence map as input. The confidence map provides information about valid and missing pixels to the normalized convolution layer, where the missing values are interpolated from their valid vicinity. Afterwards, we propose a confidence propagation criterion that allows building cascades of normalized convolution layers similar to the standard CNNs. We evaluated our approach on the task of unguided scene depth completion and achieved state-of-the-art results using an exceptionally small network. As a second contribution, we investigated the fusion of a normalized convolution network with standard CNNs employing RGB images. We study different fusion schemes, and we provide a thorough analysis for different components of the network. By employing our best fusion strategy, we achieve state-of-the-art results on guided depth completion using a remarkably small network. Thirdly, to provide a statistical interpretation for confidences, we derive a probabilistic framework for the normalized convolutional neural networks. This framework estimates the input confidence in a self-supervised manner and propagates it to provide a statistically valid output confidence. When compared against existing approaches for uncertainty estimation in CNNs such as Bayesian Deep Learning, our probabilistic framework provides a higher quality measure of uncertainty at a significantly lower computational cost. Finally, we attempt to employ our framework in a common task in CNNs, namely upsampling. We formulate the upsampling problem as a sparse problem, and we employ the normalized convolutional neural networks to solve it. In comparison to existing approaches, our proposed upsampler is structure-aware while being light-weight. We test our upsampler with various optical flow estimation networks, and we show that it consistently improves the results. When integrated with a recent optical flow network, it sets a new state-of-the-art on the most challenging optical flow dataset.Tidiga datorseendealgoritmer arbetade med täta 2D-bilder som spelats in i gråskala eller med färgkameror. Dessa är passiva bildsensorer som under gynnsamma ljusförhållanden ger en begränsad scenrepresentation baserad endast på ljusflöde. Dessa begränsningar hämmade utvecklingen av de många datorseendealgoritmer som kräver information om scenens struktur under varierande ljusförhållanden. Utvecklingen av aktiva sensorer såsom kameror baserade på Time-of-Flight (ToF) bidrog till att lindra dessa begränsningar. Dessa gav emellertid istället upphov till många nya utmaningar, såsom bearbetning av gles data kommen av flervägsinterferens samt ocklusion. Man har försökt tackla dessa utmaningar genom att förbättra insamlingsprocessen i TOFkameror eller genom att efterbearbeta deras data. Tidigare föreslagna metoder har dock varit sensor- eller till och med modellspecifika där man måste ställa in varje enskild sensor. Ett attraktivt alternativ är inlärningsbaserade metoder där man istället lär sig förhållandet mellan sensordatan och en förbättrad version av dito. Ett kraftfullt exempel på inlärningsbaserade metoder är neurala faltningsnät (CNNs). Dessa har varit extremt framgångsrika inom datorseende, men förutsätter tyvärr tät data och kan därför inte på ett effektivt sätt bearbeta ToF-sensorernas glesa data. I denna avhandling föreslår vi en ny variant av faltningsnät som vi kallar normaliserade faltningsnät (eng. Normalized Convolutional Neural Networks) och som direkt kan arbeta med gles data. Först skapar vi ett deriverbart faltningsnätlager baserat på normaliserad faltning som tar in gles data samt en konfidenskarta. Konfidenskartan innehåller information om vilka pixlar vi har mätningar för och vilka som saknar mätningar. Modulen interpolerar sedan pixlar som saknar mätningar baserat på närliggande pixlar för vilka mätningar finns. Därefter föreslår vi ett kriterie för att propagera konfidens vilket tillåter oss att bygga en kaskad av normaliserade faltningslager motsvarande kaskaden av faltningslager i ett faltningsnät. We utvärderade metoden på scendjupkompletteringsproblemet utan färgbilder och uppnådde state-of-the-art-prestanda med ett mycket litet nätverk. Som ett andra bidrag undersökte vi sammanslagningen av normaliserade faltningsnät med konventionella faltningsnät som arbetar med vanliga färgbilder. We undersöker olika sätt att slå samman näten och ger en grundlig analys för de olika nätverksdelarna. Den bästa sammanslagningsmetoden uppnår state-of-the-art-prestanda på scendjupkompletteringsproblemed med färgbilder, återigen med ett mycket litet nätverk. Som ett tredje bidrag försöker vi statistiskt tolka prediktionerna från det normaliserade faltningsnätet. Vi härleder ett statistiskt ramverk för detta ändamål där det normala faltningsnätet via självstyrd inlärning lär sig estimera konfidenser och propagera dessa till en statistiskt korrekt sannolikhet. När vi jämför med befintliga metoder för att prediktera osäkerhet i faltningsnät, exempelvis via Bayesiansk djupinlärning, så ger vårt probabilistiska ramverk bättre estimat till en lägre beräkningskostnad. Slutligen försöker vi använda vårt ramverk för en uppgift man ofta löser med vanliga faltningsnät, nämligen uppsampling. We formulerar uppsamplingsproblemet som om vi fått in gles data och löser det med normaliserade faltningsnät. Jämfört med befintliga metoder är den föreslagna metoden både medveten om lokal bildstruktur och lättviktig. Vi testar vår uppsamplare diverse optisktflödesnät och visar att den konsekvent ger förbättrade resultat. När vi integrerar den med ett nyligen föreslaget optisktflödesnät slår vi alla befintliga metoder för estimering av optiskt flöde

Ellipse Detection for Visual Cyclists Analysis “In the Wild”

Autonomous driving safety is becoming a paramount issue due to the emergence of many autonomous vehicle prototypes. The safety measures ensure that autonomous vehicles are safe to operate among pedestrians, cyclists and conventional vehicles. While safety measures for pedestrians have been widely studied in literature, little attention has been paid to safety measures for cyclists. Visual cyclists analysis is a challenging problem due to the complex structure and dynamic nature of the cyclists. The dynamic model used for cyclists analysis heavily relies on the wheels. In this paper, we investigate the problem of ellipse detection for visual cyclists analysis in the wild. Our first contribution is the introduction of a new challenging annotated dataset for bicycle wheels, collected in real-world urban environment. Our second contribution is a method that combines reliable arcs selection and grouping strategies for ellipse detection. The reliable selection and grouping mechanism leads to robust ellipse detections when combined with the standard least square ellipse fitting approach. Our experiments clearly demonstrate that our method provides improved results, both in terms of accuracy and robustness in challenging urban environment settings.Funding agencies: VR (EMC2, ELLIIT, starting grant) [2016-05543]; Vinnova (Cykla)</p

Propagating Confidences through CNNs for Sparse Data Regression

In most computer vision applications, convolutional neural networks (CNNs) operate on dense image data generated by ordinary cameras. Designing CNNs for sparse and irregularly spaced input data is still an open problem with numerous applications in autonomous driving, robotics, and surveillance. To tackle this challenging problem, we introduce an algebraically-constrained convolution layer for CNNs with sparse input and demonstrate its capabilities for the scene depth completion task. We propose novel strategies for determining the confidence from the convolution operation and propagating it to consecutive layers. Furthermore, we propose an objective function that simultaneously minimizes the data error while maximizing the output confidence. Comprehensive experiments are performed on the KITTI depth benchmark and the results clearly demonstrate that the proposed approach achieves superior performance while requiring three times fewer parameters than the state-of-the-art methods. Moreover, our approach produces a continuous pixel-wise confidence map enabling information fusion, state inference, and decision support

Uncertainty-Aware CNNs for Depth Completion : Uncertainty from Beginning to End

The focus in deep learning research has been mostly to push the limits of prediction accuracy. However, this was often achieved at the cost of increased complexity, raising concerns about the interpretability and the reliability of deep networks. Recently, an increasing attention has been given to untangling the complexity of deep networks and quantifying their uncertainty for different computer vision tasks. Differently, the task of depth completion has not received enough attention despite the inherent noisy nature of depth sensors. In this work, we thus focus on modeling the uncertainty of depth data in depth completion starting from the sparse noisy input all the way to the final prediction. We propose a novel approach to identify disturbed measurements in the input by learning an input confidence estimator in a self-supervised manner based on the normalized convolutional neural networks (NCNNs). Further, we propose a probabilistic version of NCNNs that produces a statistically meaningful uncertainty measure for the final prediction. When we evaluate our approach on the KITTI dataset for depth completion, we outperform all the existing Bayesian Deep Learning approaches in terms of prediction accuracy, quality of the uncertainty measure, and the computational efficiency. Moreover, our small network with 670k parameters performs on-par with conventional approaches with millions of parameters. These results give strong evidence that separating the network into parallel uncertainty and prediction streams leads to state-of-the-art performance with accurate uncertainty estimates

Propagating Confidences through CNNs for Sparse Data Regression

In most computer vision applications, convolutional neural networks (CNNs) operate on dense image data generated by ordinary cameras. Designing CNNs for sparse and irregularly spaced input data is still an open problem with numerous applications in autonomous driving, robotics, and surveillance. To tackle this challenging problem, we introduce an algebraically-constrained convolution layer for CNNs with sparse input and demonstrate its capabilities for the scene depth completion task. We propose novel strategies for determining the confidence from the convolution operation and propagating it to consecutive layers. Furthermore, we propose an objective function that simultaneously minimizes the data error while maximizing the output confidence. Comprehensive experiments are performed on the KITTI depth benchmark and the results clearly demonstrate that the proposed approach achieves superior performance while requiring three times fewer parameters than the state-of-the-art methods. Moreover, our approach produces a continuous pixel-wise confidence map enabling information fusion, state inference, and decision support